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* ( « * The paper describes the major evaiijation problems 
-facing early intervention projects and the solutions for these 

problems ^adopted by the PVeschool Program, Center on Hunan 
Development, which serves handicapped and nonhqndicapped childxen 
(birth to- 5^ years) and' their families at the University of Oregon. 
Five components of. the program are briefly outlined: intervention in 
the classroom and in the hone, parent involvement, support services, 
training and dissemination, and evaluation. Difficulties in ' 
evaluation are considered for fou* categories: resources for. / 
evaluation, framework for guiding evaluation efforts, suitability of 
existing ieasures, and* appropriateness of available analytic designs4 
Among the solutions offered are - the following: whenever possible, 
collect data whiph have, multiple purposes; after identifying the f t 

* goals k of the program, spied instruuents that reject the program's 
emphasis; when nonstandard instruments are available that match 
program goals, define in clear behfeioral terms what those goals are, 
then,.use those ^definitions a£ the basifc. for developing an assessment 
measure; and evaluate child progress using multiple indites and . 
conditions .when .possible. Outlined is -a, plan m for documenting child 
progress in t&e Preschool Program the. Center on tinman Development. 
A plan_ is outlined for documenting Shild progress in terms of 
resources, framework, assessment measures, data analyses,., data 
preparation, simple data description, and estimation of program 1 
effects. Tables and figures provide Information on instruments v used 
in the, evaluation plan, testing schedule, And training targets. Among 
the tables and figures provided are a^ description of instruments used 
in the evaluation plan, sample recording forms, aad flow charts of 
important considerations in instrumfen-tt selection, data preparation, 
and data analysis. Appended materials include descriptions* of 1 
assessment instruments, a sample .demographic form, and additional , 
analytic strategies for assessing program impact. ,(SB) 
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Educational programs and allied health services for handicapped infants and preschool 
children have grown increasingly available during the past decade (Bricker, Seibert & Scott, in 
press)* In large measure, this growth in services is due to federal legislation such as 
P. L~ 9*- 1*2, which provides states with modest incentives for serving preschool handicapped 
children until 1982, and, tQ state regulations Establishing educational services for young 
/ Aandicapped children. Cohen, Sefnmes and Guralnick (1?79J report that by 1979,-31 states had 
some form of mandated special education services for children ugder age six* 

Although federal and state legislation guarantees young handicapped children the right to 
early education, continued support from both federal and state governments is dependent in 
part upon the ability of early childhood special educators to demonstrate that early , 
intervention has significant educational impact upon young handicapped children. If special 
educators cannot objectively demonstrate that early intervention produce^ long -range, benefi- 
cial differences in handicapped children's educational achievements, public support for early M 
intervention programs stands to be severely reduced or completely eliminated. EE special 
educators,, however, cSn show that early intervention programs produce a significant facilita- 
tive impact on young handicapped children, public support is likefy to continue, at least at 
marginally sufficient levels. ^ * " - 

Lack of objective support lor the effectiveness of various educational strategies being 
used with young handicapped children is a serious problem (Bricker, 1978* Clarke & Clarke, 
1976). The lade of empirical verification of program impact can be attributed to several 
factors, including inadequate assessment instruments, inappropriate designs and impractical 
documentation methodologies (Sheehan^l979, 1980). JTbe absence of a framework to guide the 
documentation of child progress farther complicates the problem of objective demonstrations 
of effectiveness. Thus, an urgent , nefed exists for appropriate and practical evaluation, 
strategies. This paper is directed toward this need and describes an early intervention 
program 1 * comprehensive evaljjati^n plan which was focused on document ihg* child progress. 
While we recognize that a complete evaiu&tioo plan should also address not only such areas as 
child progress, but parent involvement, staff tr&ining.atfd other important activities, evalua- 
tion of these areas is unfortunately, less refined thah strategies for monitoring child change 
and therefore remains a major challenge for the eighties* 

This paper describes the major evaluation problems facing early intervention projects 
and thi solutions for these problems adopted by the ^reschodt Program, Center on Human 
Development, at the University of ^Oregon, Eugene. Rather than foqusing on the quantitative 
results of the program, we attend primarily to the manner in which the program addressed the 
issues raised in the discussion of evaluation problems*' We hopc^ that this approach will provide 
a useful model for other intervention programs" facing the task of evaluating" program impact. 

4 

* * 

* Program Description 

• * • * * 

Because of the diversity of its target population and the congruence of its programming 
with other pearly childhood intervention programs, we have selected the Preschool Program at 
the University of Oregon as a useful model ffom which tq discuss evaluation. The Preschool 
Program served handicapped and , no nhandi capped children ranging in age from birth to fivje 
years anct their families, the £hildren,'s handicapping conditions evolved from numerous 
etiologies and ranged from mjld to profound. Althpugh such diversity among the largef 
population tends to complicate the problems of documenting child progress, the evaluation 
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suited for such a diverse population" are more universally applicable than those 
suhe<f6Qly for a more specific population. < 

Like many early childhood intervention programs, the Preschool Program was composed 
of five separ&te bdt integrated components: intervention, parent involvement, support 
services, training/dissemination and evaluation. Each .component is briefly described here to 
plabe the evaluation plart in perspective. 1 * * • • 

The intervention cortiponfent, divided into a classroom unit and a home intervention Wit, 
afforded classroom and home, instruction in gross motor, fine 'motor, sensorimotor, social, 
self-help and communication. Instruction at the Center and in one public school classroom 
was conducted primarily in small and large groups, although individual instruction was offered 
when necessary. The basic .goal, of classroom intervention *fes to develop and implement^ for 
e^ch child the most effective educational progranj^ and each child engaged in 15 to* 20 
instructional activities interspersed with periods of exploratory piay daiiy. # Through home 
intervention, the staff trained parents to become effective teachers of their children and 
made weekly home visits to observe progress and model new parent activities. The range in 
age of children ir* both units was 4 jnonths to 5 years; their handicapping conditions ranged 
irom mild to severer Several at-risk and nonhandicapped children participated in the 
iclassroom.unit. t ~ 
1 TKe emphasis of the parent involvement component was threefold; educational, social 
Service and advocacy. In additiorf to individual and small-group instruction, the staff arranged 
wefekly and monthly parent meetings and coordinated social service counseling for the parents 
based on their needs. , Because the Stalf encouraged parehts to become articulate spokes- 
persons not only for their own child but for preschool handicappe$J"children in general, they 
supported and informed the parents about advocacy activities. 

A fffiysical therapist, psychologist, Speech pathologist, social worker arjd feeding special- 
ist were involved h> the support service component. Because this program was unable to 
suppor t-the cadre pi needed professionals on a fullVtime basis, the program adopted a model 
.cabled "the educational synthesizer" (Bricker, 1976)i* In this model r the specialist functions 
primarily as an evaluator and consultant who subsequently monitors the implementation of the 
developed program. In such a model, the classroom staff and parents rather than the specialist 
provide the primary hands-on therapy to the child. In this approach, # the teacher and parent 
are responsible for organising Ideas frpm other disciplines utto an integrated developmental^ 
sound approach. , # . J 

Th^ training and dissemination cpmponent covered all aspects of *fne program including 
evaluation, curriculum and administration. Graduate students from a variety of disciplines 
used the program as a practicum, local, state, national and international visitors came to the 
program site for demonstrations of early intervention procedures, and the program generated 
data'and information for writers of articles and reports. 

As we shall fully explore below, s the primary objective of the evaluation component was 
to document child progress. Assessment procedures fell into four categories, the collection of. 
demographic information on the children and families within the program, the pre and post 
administration of standardized or norm-referenced tests, the pre anipost administration of 
criterion-refer^r\ced test^ and the collection of daily and wefekly data on progress toward IEP 
target objectives. The remainder of this paper willdiscuss this componertt in detail. 

'Fqt detailed descriptions of project components see; Bricker, D. A handicapped children's* 
early education program. Rationale, program description and impact. Final report for the 
Division of Innovation and Devel6pment, Office of Special Education, January 1981. 



Evaluation Problems Facing Early Interventionists 

Lack of objective evaluation of the effects of early intervention with handicapped 
infants and preschoolers doe^ not result fipm indifference or neglect on the part of the 
interventionists. Evaluation of program effects is. an arduous and time -consuritin^ undertaking 
laced with many complexities and problems* These difficulties can be divided jfito four 
distinct categories of problems: J 

* * * 

1. Resources for evaluation 

2. Framework for guiding evaluation efforts \ t 
2* Suitability of existing measures * 

Appropriateness of available analytic jdesigos 

In the following sections, we will look closely at each of these categories and offer possible 
solutions to the evaluation problems inherent in each-j J 

Resources for Evaluation 

A serious problem often overlooked by critics of eajly intervention programs is the lack 
of necessary resources to pursue eyajuation activities. A sound evaluation plan often requires 

- the assistance of diagn6sticians and psychometrioans, individiials'Vitfe, research design, 
statistical and computer analysis skills, and funds for test protocols, computer scoring and 
analysis time. In addition to the need for tangible support services, there is a need to maintain 
necessary commitment of project personnel to the . evaluation of program and/or child 
progress. Successful implementation of any evaluation approach will depend r upon an entire 
staffs willingness to support data collection efforts. 

Although there are no completely satisfactory solutions to the problems encountered 
when attempting to assemble and maintain needed support services, some potentially useful 
approaches to acquiring minimal assistance are suggested below: / , 

*' 1. Contact local colleges or universities to identify persons with appropriate training. 

. They may be interested in using an intervention program as a practicum site in 

^ . return^ for consultation. In addition, graduate students with evaluation expertise 

- # may be available to assist the program* • * 

2. Determine whether rands can be specifically identified for a part-time e valuator , 
br if an evaluation specialist can be shared with another intervention program. 

3. If they df^available, use practicum students involved in the intervention program 
. who have access to computer time for a class project dealing with the program. 

4. Whenever possible, collect data which have multiple purposes* For example, the 
Uniform Performance Assessment System (White, Edgar & Haring, 1978) is used at 
the tJniversity of Oregon Preschool for programming as well as for documenting 
phifd progress. * § - * 

5. If possible, select at least some evaluation instruments or procedures that can be 
administered by classroom personnel. Instruments such as the Uniform Perform- 
ance" Afse^ment System (UPAS) and the Developmental Profile (Alpern & Boll, 

1972) were designed for -use by teachers. 

* • • • * 

Thoughtful application of strategies such as those suggested above should allow most 
programs to maximize* the ustfof resources available to collect information pertaming/to child 
progress. * * ; 
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Framework to Gukfighe Evaluation Effort 



A sound and effective educational program needs to be governed or directed by some 
broad set of guidelines 6r framework. Such a framework should provide cohesiveness and 
consistency to a program by directing decision-making in a number of areas including; 1) the 
determination of short-term and long-term .objectives afld priority areas for the target child's 
individual educatipr) program* 2) the selection of strategies for facilitating attainment of the, 
established objectives* 3) the selection of appropriate evaluation instruments to assess initial 
leyels of entry into the program and to monitor subsequent chanjjgj and 4) the^construction, 
adaptation and/or modification of training materials and curricula. 

the adoption of a framework will also assist in tjie intelligent selection of specific 
instruments and subsequent analyses to be performed on the data yielded by the chosen 
instruments. The need for an organized evaluation plan becomes clear w hen one .considers the 
implications 4 of following "hunches" or conducting analyses^ because fctey^might show some- 
thing." To avoid collecting a morass of data that do not lend themselves to the objectives of 
the program, we suggest that attention be given to the following strategies* 

1. After identifying the goals^o^th? program, select instruments that reflect the 
program's emphasis. ^ ■ 

2. \ Identify which of the selected instruments or subtests of instruments most closely 

^approximates the goals of the program's intervention. Also identify which of the 
selected instruments is the most adequate psychometrically.^ These may not be 
the 'same. 'IT the instruments differ, attempt to. relate the progranrr=specifie 
instrument to the psychometrically adeqi^te instrument through the use .of a. 
statistical procedure such as correlation. 

3. . Identify which change scotes will yield Jhe^most usefuLjnfwmation. For example r 

comparisons between fall entry scores and spring exit scores may be better suited 
in some cases, while in others quarterly change comparisons may be preferable^ 

4. Use individual subject information in final analyses only if it will provide more or 
different information than is yielded by, total or^aggregate scores. . 

5. When dealing with small samples of children (N^<6)7 individual analysis may be more 
. profitable than aggregate analysis. • « \ 

Consideration of the above points should encourage personnel to continue to focus on the 
documentation of child progress related to major program objectives. There should always be 
a clear and apparent relationship between the type of data collected, the analyses selected and 
' program goals. 1 * *" - ■ 

Suitability of Existing Measures t * 

A large number of assessment measures for nonhandicapped preschool children are 
available. Many of these measures have been reviewed (Cross & Johnstqn, 1977; Walls, 
Werner, Bacon & Zane, .1977; Hoepfner, Stern & Nummedal, 1971). In addition, Johnson and 
Kopp (undated) have compiled a comprehensive bibliography of screening and measurement 
instruments for nonhandicapped infants. Wh^te many of th$ instruments described in thes? 
compendiums were designed for identification and diagnosis, early interventionists often use 
them to document child, progress. Unfortunately, the further child performance diverges frop 4 
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psychometrically sound instrument is one with reported adequate reliability and validity 
information that has been -assembled in an objective manner. 



normal development, the less applicable for documentation of progress are such measures. 
Use of these measures is often Inappropriate because documentation of progress requires 
strategies which are comprehensive yet responsive to relatively small changes in child 
behavior, a Characteristic not found in many of the assessment instruments that were 
developed primarily for nonhandicapped children (e.g., Bayley Scales of Infant Development). 

In addition, many available measures have little data reflecting a handicapped popula- 
tion's performance on the instrument. Having data only on nonhandicapped children is 
appropriate if an instrument is to be used to identify children with possible problems, or to 
diagnose the existence of a problem. However, to chart . meaningfully the progress of 
handicapped or at-risk children, some concept of the expected rate of progress for those 
children is essential. , 

The current lack of suitable evaluation instruments has led many interventionists to 
develop "home -made" tests by taking items from existing measures. This approach seem? to 
be somewhat shortsighted. Such idiosyncratic instrument development seems likety to 
compound rather than to solve existing problems because most 'home -made instruments are not 
used widely enough to collect adequate information on their psychometric properties (e.g., 
their validity and reliability). - . • • 

Until more appropriate evaluation measures become available for use with handicapped 
populations, we suggest that the following alternative^ be considered: 

1. Have r a clear idea of the expected developmental goals of enrolled children. 
Recognize when the goal ii normal development in all areas (e.g., at-risk infants), 
when the goal is normal development in some areas and minimal development in 
other areas (e.g., motor impaired child with normal functioning' in social, corpmuni- 
cation and cognitive domains), or when the- goal may be extremely hmited 4 change 
in the behavioral repgftdire across' domains (as In the case of profoundly handi- 

t capped children). Establishing individuaLand group goals should guide the search 
for selecting the most appropriate instruments (Sheehart, 1979; HarJ)in, 1979). 

2. When possible, select instruments which have known reliablity, validity and 
* * normative d&ta. # - * 

3* Use a battery of instruments which reflect desired program goals. If possible, 
include standardized instruments in the battery to relate them to non-standard 
instruments. For example, examine the relationship between a child's performance 
on UPAS and on the McCarthy Scales of Children's Abilities (McCarthy, 1972). 

<t. When no standard instruments are available that match program goals, define in 
clear behavioral (objective) terms what those §oals are, then use those definitions 
as the basis for developing an assessment measute^ ' 

The above suggestions are compromise solutions but must suffice until the field develops 
more appropriate instruments for measuring the progress of young handicapped infants and 
children. 4 

Appropriateness of Existing Analytic Designs & 

• *" 

Use of traditional research designs confronts the early interventionist with seriOUs 
problems. Randomly assigned control groups ar^ generally not possible (Bricker, 197S). 
Quasi -experimental designs offered as substitutes for such control groups still, generally 
require comparison subjects who receive no treatment an<T*therefore are often equally 
impossible (Qjpk & Campbell, 1979, Bricker & Sheehan, 198G). Nevertheless, two alternatives 
are available to contend with the problen/ of no control of comparison groups. The first 
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alternative involves the use of analytic designs, which depend on statistical comparison groups. 
The alternative involves comparisons using other existing intervention programs. t j 

Jpe general mode! using statistical , comparison groups has been labeled either a 
prediction mode! (Sheehan', I9i0) or a norm-referenced model' (Horst, Tallmadge & Wood, 
1975; Tallmadge, 1977). In essence, this analytic mode! involves the computation of predicted 
progress for handicapped children and the comparison 9f actual progress with that prediction. 
Predicted progress can be derived in several ways such as reference to existing norms, 
correlations between age. and pretest, and correlations between posttest of one year and 
posttest of the next year. \ 

Each of these .analytic mocjcls faces problems stemming from assumptions of linear 
growth and stable estimates of progress (Linn, 1979). Growth of handicapped children .(or even 
nonhandicapped children) does not necessarily conform to those assumptions. The effects of 
deviations from assumptions of linear growth and stable progress on the analytic designs is 
unclear. Nonetheless, intervention programs should consider the incorporation of prediction 
designs, rather than rely exclusively on a pretest/posttest comparison with no estimate of 
progress in the absence of intervention. 1 » A 

v A second alternative for non-comparison controls is to compare child progress in a given 
uvtervenjion program to child progress in other intervention programs. If interventionists are 
billing *lo compare child progress across programs^ such designs are possible. This possibility 
will become more appealing as the question "Is intervention better than no intervention*' is 
replaced with "Which intervention is most effective?" 

In the development of an evaluation plan, we suggest the following strategies be used to 
deal with problems posed by design issues: • * 

1. If possible, use an -Instrument lor which normative data exist. Using an instrument 
with norms, one can derive expectations of progress on other, non-normed, 
instruments through correlation of the two instruments. For example, a compari- 
son can be made of children's performance on the Bay ley with their performance, on 
a non -standardized checklist- If a reasonable correlation is found between the two 
instruments, then one can place more reliance on the checklist as an accurate 
reflectibn of child progress. Keep in iflind, however,* that a low correlation does 
not necessarily mean, the checklist is inappropriate. 

2. Be prepafed to modify expectations of progress If the derived expectations 
consistently Underestimate or overestimate actual child progress. For example, 
historically the predictions of progress for Down's syndrome children were bleak; 
now outcomes for these children are more optimistic, and expectations must be 
.modified accordingly (Hanson, 1977). 

3. Use complex analyses sparingly. Other evaluation efforts have shown that 
statistical complexity can result in limited generalization and Interpretation 
(Haney, 197?). . 

If possible, make arrangements with other intervention programs to share data 
bases. Be sure to establish procedures for guaranteeing anonymity of children, 
• parents and projects. ' , ^ 

5. Consistent patterns of performance across tests, conditions and children lend 
validity to the general nature of the results. Therefore, evaluate child progress 
using multiple indices and conditions when pbssible; for example, include standard* 
ized tests, criterion -referenced tests and datatsfilected in different settings. 



Documenting Child Progress: An Evaluation Plan 

The deslgriof a plan documenting child progress necessitates the adoption of a pragmatif 
perspective. The remainder of this paper -describes the documentation oi child progress in the 
Preschool Program. at the Center on Human Development, University of Oregon. The 
evaluation strategies were selected as. a Balance between methodological purity and the 
demands of field research In a service delivery system. The detailed description of this 
evaluation plan for child progress may serve as a useful model, m whole or in part, for other 
early intervention programs. The evaluation plan despgted below is composed of four major 
sections that are consistent with ^he problem arearf^previously discussed. The four sections 
are; resources, framework, measures and analysjfSf ■ 

Resources « 

Location of this program in the Center on. Human Development has fostered the 
combination of a variety of resources to benefit current research, training and demonstration 
efforts. 4 For example, providing graduate training in the area of early childhood special 
education necessitates placement of students in site^ involved in the active delivery of 
educational and allied health services to handicapped^ infants and children. Preferably, the 
staff at these sites are engaged In effective intervention approaches as well as in investigating 
innovatiye strategies which may ultimately improve intervention efforts. While the demoiv 
stration program provides students with practical experience, the students in turn provide the 
program with the valuable res6urce of an additional workforce to direct individual and group 
activities for the enrolled children. ; 

The research activities have "been defined to include^ documentation of child progress as 
an important goal, therefore, rather than operating the research project independently of the 
demonstration unit, the demonstration unit h?s become one of the focal points for the research 
staff. In particular, the research staff has assisted in preparing the outcome data for analysis 
and in conducting subsequent analysis, largely using prepackaged computer programs. 

'The state agency, Crippled Children's Division, which is primarily responsible for 
evaluating and diagnosing develop men tally disabled children in Oregon is also housed in the 
Center. This proximity has enhanq&d a cooperative liaison in which the CCD completes the 
initial evaluation and diagnostic activities on a .child, leaving the intervention staff free to 
focus their evaluation ef forts^n educational assessments. 

Framework 

The theoretical framework we have developed for evaluation is Based on three tenets: 
1) behavior changes in form from the simple to -the more complex, 2) disequilibrium produced 
by changing environmental demands, is necessary for eliciting new adaptive responses, and 3) 
behavior develops sequentially following general but consistent guidelines. 3 



^We are well aware that handicapped infants and children .often show behavioral deviations 
produced by a*speafic disability ('see Fraiberg, 1968, or Kopp & Shaperman, 1973). These 
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This developmental-interactive approach, a combination of developmental theory and 
behavioral technology, provides a general map of emerging behavior* These maps are based on 
data generated by the developmental literature in the domains of motor, sensorimotor, 
communication and social behavior. Such developmental hierarchies. are composed of a series 
*of sequentially acquired behaviors and currently provide our best , criteria for establishing 
objectives for child programming in all but the most severely handicapped individual* This 
framework not only allows us to specify long-term objectives but assists us in determining the 
most probable sequence for successful programming. 1 • * ] 

9 T7>is developmental-interactive framework with' its attention to developmental requisites 
or concurrent responses can suggest what the immediate intervention priorities should be. 
Sensitivity to relationships among domains of behavior can help the interventionist select 
targets that, are appropriate both within and across domains. For example, many e;arly social 
and self-help skills require a level of understanding oJLpJyects in terms of their social utility 
(e.g., spoons are for eating, shoes are for wearing). To understand th^se functions, \t appears, 
that the child must have passed the stage whefc objects are only sucked, banged or, dropped. 
The child must be alert to the unique physical properties of objects tiefore he or she can begin 
to understand their social significance* v v \ 

The develop mental-interactive framework indicates strategies ofunjerventioh £hat will 
foster development toward the chosen program objectives. Beyond suggesting a task analysis, 
ttfe information available suggests strategies wt)ich should, if applied correctly, produce more 
functional and universal response forms. For example, the interventionist operating within 
this framework will not focus exclusively on verbal expression when wvc kmg on communic^itidn 
but will also target important sensorimotor and conceptual skills that research suggest^ 
under he or coincide with early language acquisition. This framework can also provide a basis 
for selecting evaluation instruments that specify ^easorfable developmental sequences. Such 
sequences should lay out objectives according to the processes that probably underlie the 
target areas and suggest the interrelationships among target domains as well. 

Finally, the developmental-interactive framework can direct the modifications and 
adaptations of curricular materials for individual children as well as guide the construction of 
new materials. Developmental sequences taken from ttfe literature can only provide a general 
set of bench marks. Within this general framework, the interventionist itfUstJbe prepared to 
make numerous modifications to me^t the needs of young disabled children* who may, vary 
significantly from the developmental patterns typically followed by nonhandicapped children. 
That is, a general sequence of development may suggest that infants learn to roll over, Crawl, 
pull to stand, and then take independent, steps. A motoncally handicapped child may not be 
able to crawl or pull to stand, but t^ie goal nonetheless is independent mobility for that child if 
at all possible, even though the interventionist may have to substitute some form of support 
for the child to be able to stand and walk for moving on a crawling board. J 



deviations appear primarily to be tied to the surface responses rather than to the^ concept^ 
or superordinate level. Hearing -impaired children may npt learn orai communication, but they 
do acquire the ability to send and receive communicative messages. So too motor impaired 
children may never walk without support> but most likely they will acquire some means for 
ambulating. General maps of development can provide, useful guidelines for most children , 



except, perhaps, for the most severely disabled. 
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Assessment was directed toward collection of information and monitor ing^of child 
progress in four % distinct but related areas: demographic, Inorm-.referenced, criterion-refer- 
enced and currieular. Table 1 contains the areas of Evaluation .included, the specific 
, instrument, frequency pi administration and Expected outcomes. Table 2 contains the testing 
schedule used in the program. Appendix A contains briei descriptions of the assessment 
instruments shown in Tables 1 and 2. The paragraphs which follow describe the assessment 
methods in each of the Jour areas* " ' » 

Demographic Assessment . Upon entry into the foogr&m, a demographic form (Appen- 
dix B) was completed on each child and his or her family. This brief fonn contains items which 
have been Constructed to yield quantifiable responses. Areas covered included; identification 
and description of the child and family, prenatal data, information regarding child's impair- 
ments and, handicapping conditions, and findings of previous assessments. 

Demographic forms were updated each fall at the beginning of the school year or when 
any children entered the program. The information collected from the demographic data 
forms (minus identifying information) was then transferred tq FORTRAN sheets, along with 
code numbers. The data werp then key punched onto computer* cards to be used in subsequent 
analyses. " 1 * 

Norm -Referenced Assessment . As indicated earlier, no Weal or completely appropriate 
instruments are available for use with pdffulations of handicapped infants and preschoolers. In 
recognition of this deficit, interventionists must make compromises in selecting norm- 
referenced instruments. This project, therefore* chose two iftstruments which have norms, 
available for n onh and i capped children. These include the Bay ley Scales of Infant Development 
(Bayley, 1969) and the McCarthy Scales of Children's Abilities (McCarthy, 1972) (see Appendix 
A). > ' ' . J 

The Bayley Scales and the McCarthy Scales werq administered during September and 
4 October or upon entry, and again in May throdgh June or .upoA exit. The staff psychologist* 
administered these tests according to the developmental^^ s^etfule contained in Table 2. 

It should be noted that an overlap exists irt test a4?t&Trm^S^of the Bayley Scales and 
the McCarthy Scales. With nonhandicapped children, the Tftayl^y Scales are considered to be 
suitable for children up to 30 months of age, and the Mc£arjffy Srales are considered to be 
suitable for children exceeding 30 months of age. These Jbbnsider at ions mu$t be altered for 
handicapped children, since the performance of many handicapped children can be scored on 
the i Bay ley Scales after 30 months of age, when performance cannpt be fc scored on the 
McCarthy Stales^ The staff of the intervention program decided to administer the Bayley 
Scales until a child functioned at a developmental age of 30 months* 'At this point, the 
McCarthy Scales were also administered until a child achieved a scorable General Cognitive 
Index (GO) on the McCarthy Scales, at which point 4the Bayley Scales were no longer used. 
TKis procedure resulted in a smooth transition in documenting handicapped children's progress 
between the developmental periods measured by the Bayley Scales and the McCarthy Scales. 

The norm-referenced data were coded On FORTRAN sheets according to a specified - 
format- Computer entry was directly derived from test protocols to avoid error. 

Criterion-Referenced Assessment . Three criterion-referenced instruments were used in A 
the intervention program. These included the Student Progress Record (SPR), the Uniform 
Performance Assessment System (UPAS) and the Adaptive Performance Instrument (API) (see 
Appendix A). Differences between norm -referenced assessments and criterion-referenced 
instruments are not always clear (Tallmadge & Horst, 1976); however, there are distinctions. ' 
Criterion-referenced instruments generally have more items than norm-referenced instru- 
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Table t 

* | ■ 

Description of Instruments Used in the Evaluation Plan 



* * *. 

Areas of • 
0 h m Evaluation 


1 *" 1 

* Instrument 

- < 


a , ' t 

Frequency of • 
_ Administration, ✓ 


• * * 

Outcomes 


• 

Demographic Information 


PrescHool Demographic Form 
* • * 

■ '. .v •• •'• , 


Entry in^ program* Beginning of 
year; Exit from the program 


' "T 

i - * 

, Quantifiable informa- 
tion on ^hlldan^ family 


ftorm Re*f ei enied Data 

( 

« 

* 


* * «■ a 

Bajriey Scales of Infam Development 

V 

« McCarthy, Scales of Children's Abilities 
(McCarthy, 1972) / 

*c 


} r i , ' 

« - # 

+ Emiy lntg p<ogiarn> Begmninjand - 

^env^oi scnooi yearj cxii ironrrtne 

"program < * *' 

• * 

■ •- - •> 


i 

Raw scores, Scale scores 
Developmental ^uoiien** 
-■or Age Equivalent 

• 


Cnterton-RefeiefllV* 
Data 

" ' 't 


Student frogress. Record ^Oregon State 
Mental Health Division, 1977) 

Uniform, Performance Assessment System 
(White et air, 197S) 

• •* - - i — ■- — 
Adaptive' Performance Instrument *. ' 
"(CAPE, 1974) , „ 


Beginning and end at sohodi yeai 

...'/.■ 
Quarterly 

• 

• — <— * — — 


Percent of items passed 

% 


Quarterly 




' * * *r 

. Curriculum -UnHed 
Data 

• * 


Preschool curriculum<covering the 
following areasi , 
* CommuAica'tlon ? A 
, Fine and Gross Motor*, 
■ Sensorimotor or Preacademio/ 
SocUI/SclNhejj) , 


Weekly probes s 
* • 

f 

a ^ 


Individual progress data 
toward objectives. 

a . ' 



t 



/ • 



Table] 
. Testing SdwdiJe 
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Instrument 




Developmental Age 








* 


Birth-2 years 


2-3 years 
*'• 


3*# years 


k 

0 years 


■ 


f 

• Bayley | 




i 


• 






McCarthy - . 

• * 
■ 

* 

\ 

Adaptive Performance - a ' 
Instrument 


r *- - 


* 




• 

• 


- 






■ 


• - 


m 


Uniform Performance 
Assessment System * 




* 




* ~d 


f 


\ 

■ < -.. V 


• • 




j. • - . 


t- * 
■ • 


Student Progress 
Record 


•' 








ft < 




* *" 






t 


' durriculum Monitoring _ • 




ft 

■ 




• • 


% 












" ^^^^^^^^^^^ 




* 

* » 


ft 


• 










* 






• • 

* 

• ■ 

• • 
Ik 

14 ' ' 


7 - 
• 

• 


• 

f 


• 

V 


► 

• 

• 
» 


M 

15 '. 


* + 


1 


* 




• 


M 



ments which tend to -make chter ion-refetenced approaches comprehensive rather than 
representative* Further, the*cbnteKT»f criterion-riferenced instruments often more clqseiy 
approximates the objectives of inter V&tion than does the content .of. the norm -referenced 
measures. Items in criterion-referenced, developmental iristruments are generally chosen 
because of their apparent value as educational training targets^ Items in the norm -referenced 
instruments are chosen for their predictive value, their presumed importance in normal 
development, and their ease of administration. Norm-referenced items are not usually chosen 
for specific educational relevance. 

Criterion-referenced m^zfsares provide norms or points of comparison, just as the 
norm -referenced measures do,. For example, two of the three instruments noted, UPAS and 
SPR, provide comparative. data* the third instrument, API, is currently being field tested, from 
which some initial standardization data wilt-be^forthcoming. Establishing norms is not the 
major advantage of the criterion-referenced measures; the strength of these instruments lies 
in the relationship which tHeir pogtent has to specific program goals and objectives. 

. The Student Progress Record (SPR) is £ developmental^ based instrument that covers 1* 
important ,areas o^* behavior (Oregon State Mental Health Division, 1977). The teacher or 
home interventionist administefed*this test in the fall and again in the spring. The program's 
instructional staff administered the Uniform Performance Assessment System (White, Edgar £c 
Haring, 1978) and the Adaptive Performance Instrument (CAPE, .1978) quarterly according to 
the developmental age schedule specified in Table 2. 

Cufricular Assessment . In the fail; an IEP was written for 'each child in which parents 
and staff specified tong-terrri i^and quarterly short-term goals to be used in curricular 
assessment.. Ta insure a cdmprehensive intervention program *for each child and to insure 
continuity for a child from year to year, a set of programmatic training targets were 
deVeioped for the following domaw^. Janguage, speech-hearing, gross motor, self-help,, 
pre-academic, fine motor *rttJ sensorimotor. Each domain was divided into a number of 
sequential targets that were appropriate W the majority of children enrolled in the program. 
The, targets were laid gut developmental ty. and as often as possible, the intervals between 
targets approximated two* to three mont£jkA& an. example, Table 3 contains a list of the 
long-term training targets, birth through 36 months, for the fine motor domain. 

If an infant were developing normally, one youid expect that ail five fine motor targets 
specif^ from birth to 12 months in Tabled would be accomplished during the first year, while 
mildly » moderately handicapped infant would be expected to acquire fewer targets, and the 
mgre severely impaired infant, still fewer targets per year. The same procedure was used to 
determine long-term targets across domains. Following each school year, a new IEP was 
written for each child mcprporating a new.set of t long-term training target?. If the infant, for 
example, had completed the first three targets in the fine motor area, items 4 through 7 might 
be included as targets for the CPming year. Such a system provided a general continuity for 
the selection, of IEP objectives from year to year for*each child, After tafgets wjere selected ^ 
and arranged by priority, 3 planning sheet (Table *) iot the fhosen targets was completed. 

To insure consistency and continuity m monitoring child progress, systematic, data- 
collection activities were necessary. Jo assist the instructional staff, three types of 
data-recording forms were devised. These forms were designed to collect tnal-by -trial data, 
multiple targets or subjects data and continuous data. 



Table 3 

Training Targets for the Fine Motor Domain 











. \ 


* 


Age in 
Months 


a * ■ * 

^ ■ Training targets * 

- 

a 


r - 


• 


1 

r 


Skill 


* 

^ Activfty 


i 


1 












0-12 * " 


Pu* tt-* rsav wl aj im 1 1 w <fii ay 1 awl 
* 

\ VohJBtar / release 


* \ 

1 TrA*4r« t%l\ Am mrm* t K/vrh rf irjj~tljtrn 
la lilOQ lOv {K£TCC3y WUl vUCVfclUn>*, 

2. Grasps objects held at all kveU- ' 
3* Releases block into cup. 








Midline orientation t 


Manipulates objects with both arms 






t 


a* * 

Transfer of object 


about midline. + * 
5a Uses both hands to transfer objects* 
■ 


X. 


m 


12.2* 


Pincer grasp 


6. Picks up items with pincer. 


\ 






Wrist roUtlon # - 


7a Turns lid on jar; opens door. 




• 




Refined pincer grasp 


f. Turns pages in book. 


< 






Eye-hand coordination 


* .9. Builds tower of 2-3 blodtt* 








Visual Motor " * * * 

* • 

* 


10a Places 3 shapes in form board. 

i * ■ 




* 


a a 


EvftaJmvi ttvvH initial ana Bincer craso 
Eye-hand coordination and pincer grasp 


■~ 

J 1. Places 2 small pegs in peg board « 
12, Imitates vertical and horizontal lines. 








t E^e-hand coordination and pincer grasp 


13. Imitates circular lines. 








Visual Motor \ 


U. Uses scissors to snip paper. 




#■ 


» # 


'Visual Motor 

/ 


15- Completes 3-plece puzzle* 

) 




« 

* 


a 

• 

1 

* 

r 

* 


* 

a 

♦ 

* > 
a 


a * 

■ 

• 

** 


V ■ 

t 

I 
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Table ♦ ; • 

* ■ 

A Sample PUnrtir* Sheet 



PUn Sheet number ^ 

- r * 

PupH (chltdH rume> 



Target (area working in) 



of 



Date 



PLAN SHEET 



STQBJ (refevanr thor t-term objective!. 



/ STO completion, date (protected) 



Mana^r 



(person Implementing protf am) 
ITQBj (relevant lonjt-term oblectlve) 



LTO completion date (protected) 



Acceleration 



Deceleration 



Antecedent 



Response 



Consequence 



Response 



Conscience 



(This mciudei events set up to 
bring about objective and what 
the teacher needs to do and set 
up before she can ask for the 
targeted behavior.) • , 



(This is a description of 
the targeted behavior-* 
behavioral)? written with 
criteria specified.) 



(This is a description of 
Events set op to conse- 
quate the targeted 
behavior.) 



(This is a description of 
behaviors that are incom- 
patible with the targeted 
behavior and that may 
Interfere with the 
teacher's implementation, 
of the program.) 



(This is a description of 
events that are set op to 
move each incompatible 
response closer to the 
targeted behavior.) 
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Table J 



Sample Recording Form for Trial by Trial Oau 



Target Area Fine Motor 



Original STOf iniant will track from mid-lme to right 

■ . : ;> 

• * _: l . 

»• 



Proytrf Objective) \nt*M will track 30*. 60* . tM ^Q 1 fro?" mid -line to 



t 

LTO; Infant will track a ilowjy moving object ISO 1 In both direction!* 



Data Collection 4 . 

Specification*! Ten trlafc per day> criteria S/10 trials correct for two conse- 



cutive days 



Child! 



Teacher* 3*>« 



Date 



\ Step- 1 
Tracki30*to right 



Step- 2 
Tracks 60* to tjfeht^ 



Step- 3 



Tracks 90' to right 



SUp-t 



Comments 



6/* 
6/6 
6/7 

6/9 
6/1? 
6/13 
6/U, 

6/16 
6/17 
6/U 
6/19 



10 



♦ ♦ ♦ ♦ 




• ■ r 

' The data-recording form contained in Table 5 was designed to collect tflal-by -trial 
data and alsq to providence for indicating the fallowing information: 

a ***** 4 

Target Area % the general area within which the objectives are 

located, such as gross motor 
* 

LTO ■ the long-term objective 

Original STO the original short-term objective from the Program- 

ming Training Targets (in the event that a teacher 
giust branch from the STO in the skill sequence, this 
jspace specifies that STO from the sequence. In tt^s 
way, the data records always relate to the core skill t 
, sequences, in spile of nefcessary Variations by each 

teacher.) ( ' 

Pfogram Objectives * either the STO from the skills sequence (if it was 

appropriate), or the objective that was branched from 
the original STO 

I Data Collection the specific data-collection procedures: e.g., 15- 

minute time sample, frequency data 

* * 

This data-recording form is composed of three sections: a space to indicate the data, a series 
of columns to record trial-by-trial data, and a space for comments* Data columns are divided t 
into four sections, each composed of ten^quares. The space above the recording squares is for 
specifications of the behavior, cues and criterion for that particular target step in the training 
program. Moving from left to right, steps targeted should more closely approximate the 
program objective. This form can be used to record a variety of date, such as frequency, 
correct/incorrect, raje or interval. The bottom of the form can be used/to graph the da*a. 

Table 6 contains a form for recording data on multiple subjects or multiple targets for 
ons* child. This form is particularly useful for collecting simultaneous information on small 
groups of children or for comparing an individual child's progress across four different targets. 
If a teacher wants to increase the number of trials .per sessioq, the form is easily modified. 

The recording form contained in Table 7 was developed to afco/nipodate the 'collection 
of continuous-data in which it is important to specify the context, antecedent and conse- 
quences for a targeted behavior, as when* mortitoripg the occurrence of productive language 
, during a free play period. x m * 

In .general, data were collected on a child's progress toward targets on a daily or weekly 
basis. The frequency of data collection was dependent upon a number of factors. 

Analyses 

,The majority of analyses conducted on the^data generated from this program involved 
the use of the University of Oregon's, computer facilities. Use of computers for data an§lysis 
requires that care be taken to protect the confidentiality of children and families served by 
the intervention project. To insure privacy, code numbers were assigned to children and 
families, and these code numbers were the only identifying information computerized. A 
roster of code numbers with identifying informatidh was kept in a securf location. This roster 
had only restricted access. 
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^ Sample Recording Form for Multiple Subjects or Targets 



Name? , A f 



Name: B 



Target: Answering questions appropriately during small Target: same 
group time 1 ~ 



Steps: fitting in chair; 2 Attending to teacher; 3 responds $tcpi5 
appropruiteirwhefr4sked a Question. \ 



s 
























o 


■H 


CM 






, t. 


r- 


CO 






CM 






C0 H 


■H 


fH 


pH 








•H 






































IP 


















to 








5 


5 
























>* 


























3 


3 < 
























2 


2 / 










































i s 








0 


0 


0 * 


0 


0 


0 


0 


0 


0 


0 


0 


0 


Baselines: 































o 


<-t. 


CM 




in 


10 




CO 


CM 






"CO 






f-H 




•— 1 




•— 1 






£ 






















to 




to 


tO 


LO 


10 




to 


to 


to 


10 


to 














(5)- 








0 


* 4 






















3 3 








3 




' V 




* 3 




3 










2 




2 




2 




2 














V 










1 1 . 








1 




1 




1 




1 p 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


Baselines: 























-r 



Name! C 
Target: same 



Steps: 













O 


f-H 


CM 




in 


v0~ 
















CO 




•— 1 


r* 




. •— 1 




f-« 


f-H 






















. 












to 


to 


10 


10 


-O 


so 


to 


so 


10 


t0 


SO ; 


so 






00000000000 
Baselines; 



Circled number indicates number of correct responses* 



Name; P 



Target; same 



Steps; . 



Baselines. 

















V 








I CO 


CM 


ro 




CO 


•— 1 






f-H 




f-H 


f-H 


•— 1 




























10 


to 


so 


to 


to 


to ' 


to 


to 


to 


to 


to 
















7 














■ 


















* ' 




































3* 












3 












2 




2 




2 

< 




2 
















1 




1 








5/ 


0 t 




0 


0 


0 




? 


0 








ERIC 



■J8 



Table 7 \ 
Sample Recording Form tor Continuous Data 



* Domain* Communication 



LTOi Expreisei ttftgubtic functions of labeling of objects 



Child? A ' 



Teachert Dan 



Program Objective* responds when asked to label object 
C 



1 s 



n 



TT — 

C^S^QUENCE 



DATE 



CONTEXT 



♦ANTECEDENT 



Vocalize 



Gesture 



Word 



BEHAVIOR 



E 1 

Peer takes ball 



1/20. 



Play area/peer 



Peer points 



ball 



1/20 



Play area/peer * 



Ball rolls away 



So 



Peer looks at ball ' 



1/20 



Play area/peer 



Ball rolls away* 



points 



ball go 



Peer looks at ball 



1/20 



Plajf area/peer 



Ball rolU away 

/ 



points 



ball go 



Peer retrieves ball 



1/20 



Play a*ea/peer 



Peer holds ball 




ball 



/ 



Peer gives ball to 
Kevin 



Data Preparation 

One problem thflt plagues any complex data-gathering effort involves errors occurring at 
the time of assessment, during transfer to coding sheets, during key punching and during 
computation* .Prior to the formal d?ta analyses, the data, collected by the project personnel 
were checked for possible error. Because this project did not have the necessary resources to 
double-check all data collection and processing, the following strategy was adopted to insure 
that data were free from errors whidT might distort the analyses. After the data were 
keypunched and verified by the keypunch operator, each variable was processed through the 
FREQUENCIES PROGRAM of the Statistical Package for the Social Sciences (Nie, Hull, 
Jenkins, Jteinbrenner & Bend, 1975). the frequency analysis produced by the SPSS program 
was visually examined to determine whether the range, mean and general distribution patterns 
were reasonable, given the nature of each variable. Variables with possible errors were 
identified and rechedced. * f 

Following these preliminary checks, a computerized list df all cases (subjects) for each 
variable was gettt^atetf using the LIST CASES PROGRAM of SPSS. The list case tables were 
used to identify cas^s with questionable data. These data were checked and errors correct £d 
as necessary. Such 4 procedure may need to be repeated in order to identify all possible 
errors. s \ 

Simple Data Description v , 1 ^ 

The next analysis^nndcctaken focused on generating descriptions of the data. Using the 
LIST CASES routine of SPSS, tables of the raw data for each child and aggregated summaries 
of means, standard deviations^r^nges, etc*, were generated for the total group and subgroups 
of normal, at -risk, mildly handicapped, moderately handicapped and severe!/ handicapped 
children. Such tables can be compiled by hand if computer facilities are not available. 

This descriptive analysis was conducted to present a case-by -case description of all the 
relevant data with accompanying group statistics to obtain a visual estimate of the extent to 
which the group data were consistent with the individual data. When children"? individual data 
are not similar, or at least pot evenly distributed, those data should not be aggregated. It may 
be apparent from a descriptive analysis, for example, that approximately one-half of the 
children are regressing, while the other half are progressing. If this is so, each subgroup^hould 
be analyzed separately. Otherwise, a simple averaging process wiJIJndicate no change, clearly 
an inaccurate, indication of the actual performance of children in the program. Although this 
process is time-consuming, it is essential that case-by -case visual examination of the data be 
conducted to determine whether subsequent, subgroup analyses are needed and appropriate. 
Our analysis r£y&j^6d no such discrepancies, so data were analyzed for total groups and 
relevant subgroups * , ■ V 

The simple data-description step in the .analytic process is also useful for another 
purpose. At this juncture, the group statistics of mean, range, median.and standard deviation 
can provide an initial look at average performances at each time period. (These statistics can 
be generated by the SPSS program, CON DESCRIPTIVE.) Due to the possible need for .subgroup 
analyses, the average performances rpight be somewhat distorted, but where such analyses arc 
appropriate, they can be examined" and reported. (Any distortions in .distributions, \ such as 
skewness, are also indicated by the SPSS program CON DESCRIPTIVE-) 

This simple data-description procedure provided a perspective which, was useful in 
guiding the staff in conducting further analyses. Decisions were made to reduce the quantity , 
of data used in subsequent analyses. For example, no further analyses were conducted on the 
subtest scores of the McCarthy. Such decisions should be made when a review of the analysis 



reveals/that the »ibtest cscores «fihibit Uttle variation frorft each other or fail to "contribute 
. additional information? Subtests or total -scores can also be identified which exhibit so Uttle 
variation thatJit is unproductive to perform further analysis. The presence of strong ceiling or 
„ basal effects is another reason to forego further analysis- Ceiling and basal effects occur 
when scores cluster at the high or low end of a scale. Such clustering obscures possible 
differences between scores which would have been found if the scale's range had been greater. 
In pur analysis jf the children's performance on the B^yley. Scales, for example, we found that 
a sizeable proportion of the subjects scored below 50. Scores of this nature prohibit computing 
a Mental Development Index orj Psychomotor Development Index, and therefore we did not 
.attempt .these comparisons with/Our population of handicapped infants, and toddlens. 

Estimation of Program Effects ' 

Prograiti effects on the children enrolled in this project were determined ir\ three' ways? 
The first method was to compare pretest (fall) scores to post test (spring) scores, using a 
correlated t-test * Wdeternpin* whether there was a statistically significant difference. 

A second 'strategy involved determining whether the differences between pre and 
posttest scopes were educationally significant. Establishing educational significance, can be 
a<^hieved in two ways* Abt Associates (1977) and Tall mad ge (1977) have suggested** procedure J 
in which gains from pv to pQsttesting are compared with the average, pooled standard 
deviation, o£ the pretestiand posttest scores. If they exceed.. 25 (Abr Associates) dr,.33 
(Tallmadge), the gains arepinSidered to be educationally significant. In addition, educational 
significance can be established through a discussion of the relativ^meanings (as determined 
through the literature) of the magnitude of the obtained mean differences. For example, the 
s average difference occur during a 6-month interval may be 8 months gam. A review of 
the literature may reveal that such gain occurs* only rarely m a" specific subpgpulation of 
tj^ndicapped children^ thus, these 8-montfc gains would be educationally significant. 

The third method chosen 4o examine program effects involved comparing gams made by 
this population of children on the SPR with those made by a state-wide.control group. The 

^ * s > 

* 

*It is useful to describe the roles that inferential and descriptive statistics sJiouldj>iay in the 
documentation of intervention effects. Pre/posttest differences are commonly assessed with 
the correlated tMest. -This test is designed to determine whethpr differences found between 
related samples drawn from 5 large population indicate differences in their respective 
populations. Unfortunately, intervention prograrp/do not actually sample from clearly defined 

% populations, therefore, it is inappflc^iate to use the correlated t-test as a method of inferring 
differences in larger populations.. Inferential statistics, however, art often used to document 
differences du^ to intervention. .The hidden assumption is that, *if appropriate sampling had 
been don^, the Jesuits would have generalized to the larger population. In a sense, the results 
of inferential statistical procedures such as tfte t-t£st may be thought of as descriptive rather 
than.ir\fer£ntial., The difference is about the target group of children, not about some larger 

.population fromVhich a random sanjple was drawn. This points up the value of using other 
descriptive statistics, such as means, modg# standard deviations and the like, as ways of 
describing the target group apd how the group changed a^a result of the intervention, A 
pre/posttest correlated fltest may be a useful descriptive \ool for documenting differences, 
but since it is not generally used as an inferential statistic in intervention research, results 

^hould be generated cautiously. 
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gain scores of the projects three*, four- and five-year -old- children were compared with 
gains made over the same period of time by other three-, four- and five-year -<jld children 
attending other intervention programs throughout Oregon* A number of similar strategies are 
available to interventionists* Descriptions of these procedures are contained in Appendix C 

figure 1 presents # synopsis, of the information presented on instrument selection, data . 
preparation and anal^sn. This /igure presets an outline of the main considerations involved 
in moving from instrument selection tp data analysis. 

Summary f 

< ■ 

The Preschool Program, Center on Human Development, tias evolved a comprehensive 
and. organized evaluation plan which was initiated with the careful selection of a battery of 
instruments that included standardized and criterion-referenced measures* These measures 
were selected beca^ft^progra(n resources were .available for implementation , and, more 
importantly, because they reflected program emphasis* ^The measures were administered 
according to a preconceived plait, and the data were checked and properly stored until the 
analyses weje conflicted. Prior to analyzing the results, the data were inspected for eqror< A 
straightforward statistical analysis was adopted to compare the pretest with posttest perform- 
ance across all measures. In addition, educational significance was measured. The outcome of 
this evaluation plan has been presented in detail elsewhere (Bncker, 1981, Bncker & Sheehan, 
1980). 

Documentation of child progress is a complex task, and the field of early childhood/spe- 
cial education is still struggling to develop appropriate and usable assessments and methodo- 
logical strategies to determine program impact. 'Recognizing this deficit, this paper has been 
formulated to present an evaluation plan that may enhance the documentation efforts of the 
field. • . 
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Fkmdmt%td tapccta* C mfcfcrathw In hyfapcfll Selection, 

ad D*t* Analysis' 



Yea 



Use it. It will facilitate 
communicaticxi with other 
researchers. Standardized 
administration and scoring 
procedures Are major 
advantages. 



I. INSTRUMENT SELECTION 



Is a norm-referenced instrument 
available for your target 
behaviors?' 



No 



Is an Instrument available 
which has wide usage 
and is currently being 
normed? 



No 



Can you .afford the 
resources to develop one? 



Yes 



♦Use it. 



Yes 



Stop* You will 
sidetrack your main 
research effort. 



Collect reliability 
information* Relate 
this information to 
others you are using. 
If they overlap a lot, 
consider discarding one. 



II. DATA PREPARATION 



Yes 



Are data checked for clerical 
errors as well as test- 
administration errors? 



No 



Doyou have access to computer 
analysis packages such as SPSS 
or SAS (Helwig et a!., 1979). 

I 



Goto 



No 



Trace the flowof data back 
to the original data sheets 
to find sources of error and 
eliminate them. 



Plot the variables most central to 
your argument. Check for subjects 
who are far from the main group. 
You may want,to consider eliminating 
them from analysis (Natrella, 1963). 



Use this program to display your 
data in graphs or in sets of 
subjects such as males or females. 
Look for major discrepancies from 
expected patterns. Look for 
•subjects- with very aberrant scores. 
You may wlih to eliminate them 
from analysis (Natrella, 1963). 
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Figure 1 (continued) 
m* DATA ANALYSIS 



Yes 



Yes 



Can you toe standard statistical 
analyses? Do you have a computer 
available? 



No 



Did you randbmly sarnpfc 
from a large population? 



No 



Outline the argument to be made 
in your report prior to beginning 
detailed analyses* Focus qp the 
main points and perform the 
simplest analyses needed to prove 
your case. Doing too man/ ana* . 
lyses on the same data capitalizes t 
on chance and weakens, your case. 
For 'example, 20 t-test&on trje same 
data are; likely to produce one' ^ 
significant test at tha M level 
even it the data were generated 
from a table of random numbers. 
Appendix C outlines three useful 
strategies for demonstrating 
improvement due to intervention* 



Concentr a te your analyse* and 
reporting on simple descriptive 
measures which make the main 
points to be demonstrated. If , 
you use t-tests and similar 
inferential statistics, present 
them as descriptions of your 
groups Avoid overgeneralixing 
from your sample. Consider 
using single -subject and/or 
small sample techniques 
(Kratochwili, 197S). 



Fdcus on simple descriptive 
statistics such as means and 
measures of variability. 
These are very convincing if 
woven into a tight argument. 
Develop a descriptive picture 
of your sample with these 
measures. 
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Appendix A 
Description of Assessment Instruments 
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Bayiey Scales of Infant Development (Bayley, 1969) 

< The 'Bayley Scale is a tripartite developmental instrument designed tp be administered 
diiring Che first 2ft years of life, tha three parts are considered to be complementary and 
include: 

• (f) The Mental Scale is designed to assess sensory -perceptual acuities, discriminations 
and the ability to respond differentially to different objects and events* the early acquisition 
of '^object constancy" and memory , learning and groblebi-soiving ability i vocalizations and,the 
beginnings of verbal communication! and early evidence of tKe ability to form generalizations, 
and classifications, which is the basis of abstract thinking. Results oMhe administration of 
the Mental Scale ^re expressed as a standard score, the MDI, or Mental Development Index. 

(2), The Motor Scale is designed to measure the degree of control of the body; 
coordination of the large muscles and finer manipulatory skills of the hands and fingers. As 
the Motor Scale is specifically directed toward behaviors reflecting motor coordination and 
skills, if is not concerned* with functions that are Commonly thought of as "mental 11 or 
cognitive in nature. Results of A administration of the Motor Scale are, expressed as a 
standard score, the'PDI, or Elychor^tor Development Index. 

* (3) The Infant Beha^r Record (IBR) is completed after the Mental and Motor Scales 
have t*?en- administered. The IBR helps the clinician assess the nature of the child's 
orientations toward his or her social and material environment as expressed in attitudes, 
interests, emotions, energy, activity and tendencies-to approach or withdraw from stimulation. 

McCarthy Scales of Children's Abilities (McCarthy, 1972) 

* \ 

The McCarthy Scales were designed as,a J^pllow-up measure to the Bayley Scales of 
Infant Development. These scales are appropriate for children from 2ft to 8ft years of age. 
The test consists of 18T subtests that make Op six scales; Verbal, Perceptual-Performance, 
Quantitative, Memory, Motor ^nd General Cognitive. The General Cognitive scale is a 
composite of the Verba), Peceptual-Perforfnance and Quantitative scales. The General 
Cognitive scale has the same ( numericaKcharacjeristics as the "Deviation IQ.'\ 

.Student Progress Record (Oregon State Mental Health Division, 1977) 

* m j*J The Student Progress Record (SPR) is a developmental^ based instrument that covers 1* 
Jmportant are§s of behavior (e.g., corqjnunication, self-help). The SPR was developed in 
Oregon and adopted bf the Mental RetardatiOn/DevelopmentSr Disabilities Office as the 
statewide mechanism for monitoring progress of ail TMR and preschool handicapped infants 
and children. ♦ ^ . 

' J' 

triform Performance Assessment System (White, Edgar & Haring, 1978) 

** * 
The Uniform Performance Assessment System (UPAS) consists of a fine motor/pre- 
academic, gross motor, communication, self-help/social and behavior management scale. 
Each scale is composed of developmental^ sequenced items accompanied by extensive 
administration and criteria guidelines. A major drawback of iJPAS is the limited number of 
items covering the developmental period from birth to 2h month*. Consequently, this 
assessment instrument is useful primarily with children who are functioning above a develop* 
mental age of 24 months. 

* • • 
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Adaptive Performance Instrument (CAPE, 1978) 

# 

The Adaptive Performance Instrument (API) covers eight domains of behavior including* 
physical intact ness, reflexes and reactions, gross motor, fine motor, sensorimotor, communica- 
tions, social and self -help* Each of the eight domains is diyided into a number of test strands 
that are arranged in developmental sequences. The API concentrates on the developmental 
age span from birth to 24, months and provides procedures for novel^modifications or 
adaptations in the assessment of chTraren with specific handicapping conditions* These 
modifications are possible for children who are ''visually impaired, hearing impaired, 
visual/hearing impaired and orthopedically impaired* This instrument is currently undergoing 
field,, testing and is thus considered to be a research toot rather than a codified, published 
scale. 1 
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Center on Human Develpproent Preschool 
Demographic Data Form 



Today.'s Date 



< 



Site 



Pupil's Identification S 
Pupil's Name 



LAST 



FIRST MIDDLE 

Female 2. Male 



; ^SEX I. 

vear f 



Pupil's Birthdate 

mo day year 

Pupil's Ethnic/Racial Group 1. Caucasian 4: Black 7. Unknown 

• i— 
'2. Native Americans. Polynesian 

3. Oriental 6. Mixed 

Time of Delivery 1. Full Term (37-42 weeks) 

2. Premature (before 37 weeks) 

* ^ 3. Postmature (42 weeks or mory) 

<f. Unknown ^ 

1. 2. 3.- 5. 6. 

10. 11. Unknown 



APGAR Score at 
5 minutes 



7. 



8. 9. 



Mother's Education * 5 6 7 8 9 10 11 12 13 1* M.A. Ph.D. (Mark 

jJL * highest 

Fathgg^ducation * 5 6 7 8 9. 10 11 12 13 1* M.A.. Ph.D. level 



Other Primary 
Caregiver 

Unknown 



completed) 



* 5 6 1 8 9 10 • 11 12 13 1* M.A. Ph.D. 



Familylpcome (Annual Gross). 1. under $5,000 ft« $15,000 - $20,000 

. 2. $5,000 -$10,00t) 5. $20,000*- $25,000^ 

3. $10,000 - $15,000 6.' over $25,000 

i 

7. Unknown 

« » 
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Impairments: ^ 1 t? 

Intellectual/Cognitive: 

j !• Normal 2* J_ Mild 3. Moderate Severe 5. Profound 

6. Unknown 

J ~ 4 . 

r Hearirtg: * 

T 

* !• Normal 2.. Some Impairment 3. Deaf 4. Unknown 

Vision: \ 

!• Normal 2. Some Impairment (glasses) * 3. Legally Blind 

•4._L Totally Blind Unknown * 1 

Motor: * ^ 

!• 1 Normal 2. Some Impairment 3. Non Ambulatory 4. Unknown 

Behavioral Problems: 

,1* Normal, none 2. Mild 3.* Severe 4. Unknown 

Identified Conditions: „ 

!• 1 Down's Syndrome 6. Microcephalism 

2. Cerebral Palsy 7. Autism fj 

3. Myelomeningocele 8. None 

* • 

Seizure Disorder 9. Other " 



5. Hydrocephalism 10. Unknown 

'Standardized Test Scor^ (most recent): 

!• Bayley 4. Denver ^ 7. Unknown 

2. 'Binet _ 5. A Cattell 

3. McCarthy 6. Other 

5core: \ 

•Date: 
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Appendix C 

Additional Analytic Strategies 
For Asse^ing Program Impact 
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1) A prediction of progress can be made by computing the Expected post test scores on 
the basis of the child's pretest performance* The predicted posttest scores are estimations of 
how children would have progressed in the .absence of intervention. For example, it may be 
predicted that a subset of children will progress three months in developmental age over a 
9~ month period. The actual progress^ the children is then compared to the predicted 
progress in order to estimate a program's effects on the participating children. If the children 
progress more than three months, one-can infer that the program has had a facilitative effect* 
however, if the children progress three months or less, one cannot claim the program to have 
had a facilitative impact. Strategies similar to this are being employed in Title I evaluations 
(Wisler & Anderson, 1979), although the efficacy of this approach has been recently criticized 

_bjOJnn (1979). * . 

2) Another analytic approach .that can be employed involves the correlation (e.g., 
Pearson Product Moment) of age at pretest time with pretest score (used by Battelle 
Corporation; Stock et al., 1976). From this correlation, a predicted posttest score is obtained. 
The correlated t-test is then run between individual's actual posttest scores and predicted 
posttest scores. If the differences are significant, with actual posttest scores exceeding 
predicted scores, the program is presumed to have had an effect. This procedure may assist in 
determining the educational significance of the obtained differences. 

3) A third prediction strategy utilizes the psychometric anchor provided by the fray ley 
Scales with reference to the infant criterion-referenced measures (e.g., API, SPR). This 
approach confutes the correlation between performance on the Bayiey Scales with perform* 
ance on the other measures. If the correlation is high and positive (e.g., greater than .75), the 
developmental age scores on the Bayiey Scales can be assumed to be an accurate estimator of 
the developmental age scores of the API and the SPR. Once again, posttest performance on 
the API and SP.R cail now be viewed in the context of developmental age progress, a concept 
which has more meaning to many interventionists ar^d funding agencies.. 
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